Action: An Annotation Framework for Events in Video

MediaStreams is a system for annotating video content with semantic metadata. Because the meaning of a particular video segment is affected by segments that precede or follow it, MediaStreams supports video annotation only for semantics that survive possible re-sequencing. Consequently, the system represents action at the level of basic physical movements such as “raising an arm” or “smiling.” This project specifies how to extend primitive actions such as those described in MediaStreams to higher-level representations, up to the level that we commonly call events: noun phrases that encapsulate a connected set of actions, such as weddings, parties, and sports. Building on earlier work that describes a conceptual model for linking events to primitive actions, we show how to extend the MediaStreams metadata framework to include descriptions of event-level actions and action partonomies that show how these complex actions relate to the existing primitive actions. Our framework consists of a taxonomy, implemented as a faceted classification, and a partonomy, implemented using the ABC ontology proposed by Lagoze and Hunter. We have expressed both the taxonomy and the partonomy in the Resource Description Framework (RDF) language.